Orbital ordering in frustrated Jahn- Teller systems 
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We consider the superexchange in 'frustrated' Jahn- Teller systems, such as the transition metal 
oxides NaNi02, L1N1O2, and ZnMn2 0<i, in which transition metal ions with doubly degenerate 
orbitals form a triangular or pyrochlore lattice and are connected by the 90° metal-oxygen-metal 
bonds. We show that this interaction is much different from a more familiar exchange in systems 
with the 180°-bonds, e.g. perovskites. In contrast to the strong interplay between the orbital and 
spin degrees of freedom in perovskites, in the 90°-exchange systems spins and orbitals are decoupled: 
the spin exchange is much weaker than the orbital one and it is ferromagnetic for all orbital states. 
Due to frustration, the mean-field orbital ground state is strongly degenerate. Quantum orbital 
fluctuations select particular ferro-orbital states, such as the one observed in NaNi02- We also 
discuss why LiNiCh may still behave as an orbital liquid. 

PACS numbers: 75.30.Et, 71.27.+a, 75.10.Jm, 71.30.+h 



There is a large class of compounds containing transi- 
tion metal (TM) ions with orbital degeneracy - the so- 
called Jahn- Teller (JT) systems. Orbital degrees of free- 
dom and, in particular, their ordering or the cooperative 
Jahn- Teller effect, give rise to a very rich physics p], ||. 
The interplay between orbitals and spins often leads to 
rather peculiar magnetic structures, turning e.g. cubic 
perovskites like KCUF3 into quasi-one-dimensional spin- 
chain materials 0]. It was recently suggested that in 
some JT systems orbitals remain disordered at all tem- 
peratures, forming an orbital liquid state j|, 0, The 
suppression of an orbital ordering was discussed for ma- 
terials with a simple cubic structure, e.g. the perovskite 
LaTiC>3, and was attributed to a strong anisotropy of or- 
bital interactions and a strong coupling between orbital 
and spin excitations. One would expect to find an even 
stronger tendency to form an orbital liquid in JT com- 
pounds with more complicated crystal structures, espe- 
cially, in 'geometrically frustrated' systems 0, e.g. the 
spinel ZnMii204, in which Mn 3+ ions with the configu- 
ration {t^g^g) form a pyrochlore lattice of corner sharing 
tetrahedra, or the layered materials LiNi02 and NaNi02 
with a triangular lattice of low-spin Ni 3+ ion in the 
(t% e g) configuration. Orbital ordering in such systems 
is an open issue, interesting from both theoretical and 
practical points of view, as some of these compounds 
(LiNi02, NaNi02) are now studied as promising mate- 
rials for rechargeable batteries, and their performance 
may be significantly affected by the JT effect. 

An important difference between the TM oxides with 
'frustrated lattices' and perovskites is the angle between 
the oxygen-metal bonds connecting two neighboring TM 
ions. While in perovskites this angle is close to 180°, 
in the geometrically frustrated JT systems it is typically 
90°. Though usually ignored, this difference has impor- 
tant consequences for orbital and magnetic orderings. In 
this Letter we derive the Hamiltonian of the 90°-exchange 
and show that, unlike in perovskites, in frustrated JT 



systems orbitals and spins are essentially decoupled and 
that orbital and magnetic orderings can be considered 
separately from each other. Considering in particular 
layered materials with a triangular lattice, we show that 
the orbital interactions are strongly frustrated, which re- 
sults in a large number of disordered mean-field ground 
states. However, we argue that this frustration is lifted 
by quantum orbital fluctuations that stabilize a ferro- 
orbital state and induce a gap in the spectrum of orbital 
excitations. We also show that the ordering of spins in 
the layers is ferromagnetic. These results agree with ex- 
perimental properties of NaNi0 2 |§ and lead us to a 
conclusion that the puzzling absence of both spin and 
orbital ordering in LiNi02 || [h]] results from disorder 
rather than from the superexchange. 

We first obtain the exchange Hamiltonian describing 
interactions between TM ions with one electron or hole 
on doubly degenerate e s -levels. In this case orbital states 
are conveniently described by introducing isospins (or 
pseudospins) Tj on each TM site j, which act on the 
up and down states, | T z — ±i) identified with, respec- 
tively, d 3z 2_ r 2 and d x i_ y i orbitals. Quite generally, an 
effective interaction between the orbitals and spins on 
pairs of neighboring TM sites i and j has the form |l| 

Hst = [ J s( s * s j) + + JsTiSiSj^Tj] (1) 



(for simplicity, we do not show here the full structure of 
the orbital interactions, which are strongly anisotropic 
both in real and isospin space). In perovskites with the 
180°-bonds the spin- and orbital-exchange constants Jg 
and Jt, as well as the constant Jst, describing the cou- 
pling between orbitals and spins, are all of the same or- 
der of magnitude. This makes the spin and orbital or- 
derings dependent on each other |Q. The strong inter- 
play between orbitals and spins is also crucial for the 
mechanisms of suppression of these orderings used in 
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FIG. 1: The crystal structure of NaNi02 (a) and a plaquet- 
te in the a/3-plane (a, j3 = x,y, z) formed by two nearest- 
neighbor Ni ions, 1 and 2, and two oxygens, Oi and O2 (b). 



Refs. 11, |12j to explain an apparent absence of both 
orbital and magnetic ordering in LiNi02 ^, [To|| . In par- 
ticular, in the SU(A) version of the model Eq^Q) 0, ^ 
both the spin and orbital interactions are isotropic and 
Js = Jt = j J st- We first show that the actual situation 
in the 90°-systems is quite different: the orbital exchange 
is strongly anisotropic and Js -C Jst <C Jt- 

We derive the exchange Hamiltonian using as an ex- 
ample the layered material NaNi02, in which Ni and Na 
ions occupy consecutive [lll]-planes of the NaCl struc- 
ture (see Fig. |l|(a)). In each plane the low-spin Ni 3+ ions 
with one electron on the doubly degenerate e g level form 
a triangular lattice. The exchange between two nearest- 
neighbor Ni ions in the planes occurs via common oxy- 
gens and the angle between the two oxygen p-orbitals in- 
volved in this exchange is 90° . The exchange between Ni 
ions from two neighboring planes (separated by a plane 
of Na ions) is weak and will be neglected here. 

The electron hopping in the z-direction can only occur 
between the p z oxygen and the d,3 Z 2 -r 2 nickel orbitals. 
The corresponding hopping amplitude is denoted by t. 
All other processes are forbidden by symmetries of the 
oxygen p and nickel e g orbitals. Thus, the hopping am- 
plitude in the z-direction is t for the \T Z = +i) nickel 
state, and for the \T Z = — 5) state. More generally, the 
hopping in the a-direction (a — x, y, z) is only nonzero 
for the oxygen p a and the nickel d 3a 2_ r 2 orbitals. It is 
convenient to introduce the isospin operators I a = Tn", 
where n Q are three unit vectors in the (T x , T z ) plane 
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The operators I x and I v play the role of T z for the x 
and y-dircctions, i.e. they describe the occupation of, re- 
spectively, the d 3x 2_ r 2 and d 3y 2_ r 2 orbital. The hopping 
amplitude in the a-direction is then t for the \I a = +|) 
state and for the \I a = — 5) state. 



Figure m(b) shows a plaquette in the a/3-plane (a, = 
x,y, z) formed by two neighboring Ni sites 1 and 2 and 
the two oxygens shared by the oxygen octahedra sur- 
rounding the TM ions. An effective exchange Hamilto- 
nian for two e g nickel electrons is obtained by expanding 
the plaquette energy in powers of the hopping amplitude 



t, which is assumed to be much smaller than the energy 
A of the electron transfer from O to Ni and the Hubbard 
repulsion on oxygen U p . The orbital and spin exchange 
appears in the fourth order of the perturbative expansion 
and the higher-order terms are neglected here. 

We first neglect also the Hund's rule coupling between 
the spins of two holes on oxygens. In that case the 90°- 
exchange is independent of a spin configuration and only 
involves orbitals. The orbital exchange Hamiltonian of 
the plaquette in the a/3-plane has the form 



H T = J T T af} + const, 



where 

J a/3 = 



2 h 



2 h 



(3) 



(4) 



and Jt is the coupling constant specified below. The 
factor I — 7f counts the number of the oxygen electrons 
that can hop to the Ni site 1 along the bond in the in- 
direction. This number equals 1, if the e 9 -electron is in 
the state \If = + §), and 2, if the electron occupies the 
state \I° = — h). (For one hole on e s -level, as in Cu 2+ , 
the exchange interaction corresponding to Eq. (Q) involves 
combinations (| + I Q ) instead of (| — I a ).) The two 
terms in Eq. (^) correspond to the two exchange paths on 
the plaquette through the oxygens Oi and 02- 
The orbital coupling 
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A 2 (2A + [/ p ) A 3 A 3 (2A + U p ) 



(5) 



is the sum of the contributions of two different mecha- 
nisms of the orbital exchange. The first term in Eq.(j^) 
is due to the exchange via the intermediate states with 
two holes on one of the oxygens from the plaquette. The 
mechanism resulting in the second term in Eq.(||) works 
even for infinite U p , when the two-hole oxygen states are 
forbidden. To understand its origin consider first a sin- 
gle Ni site inside the oxygen octahedron. The hopping 
of electrons from the 6 oxygens to nickel results in the 
energy decrease SE. For two neighboring Ni sites this 
decrease is smaller than 2 SE, as the surrounding octa- 
hedra share two oxygens and the two-hole oxygen states 
are forbidden for infinite U p . The effect of the blocking 
of the two-hole states on common oxygens depends on 
orbital states of the e g -electrons, which gives rise to the 
orbital exchange with the coupling constant 

The intermediate states with two holes on one oxygen 
also result in a spin exchange. As in such states the holes 
occupy two mutually orthogonal orbitals p a and pp, the 
spin exchange is only possible due to the Hund's rule 
coupling Jh between the spins of the holes (in systems 
with the 180°-bonds the spin exchange occurs even for 
Jh = 0). The corresponding exchange Hamiltonian in- 
volves both spin and isospin operators and to the lowest- 




FIG. 2: The triangular lattice formed by Ni ions in the [111]- 
plane. Shown also is a disordered mean-field ground state, in 
which the isospins form lines parallel to the unit vector e xy , 
such that {T?) is the same on all lattice sites, while the sign 
of (T/) varies arbitrary from line to line. 



order in powers of Jh it has the form 
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(6) 



where J TS = a -2^a+u p )^ and jaP is S iven b y Ec h(!)- 
The interaction described by Eq.(||) is different the 180°- 
exchange in two important respects. First, since for 
any orbital state the expectation value (X a ^) > 0, the 
spin exchange is effectively ferromagnetic independent 
of an orbital state (the average spin-exchange coupling 
— Jts^"' 3 ) < is negative). Second, since Jh <C U p , 
the spin-isospin coupling in the 90°-systems is weaker 
than the pure orbital exchange, described by Eq.(0): 



Jts 

~j7 



2A 



Jh 



U p (2A + U P ) 



Thus, in 90°-exchange systems spins and orbitals cannot 
strongly influence each other, i.e. they are essentially 
decoupled. (A similar conclusion was reached by Reitsma 
and Feiner (unpublished). We are grateful to L. F. Fcincr 
for informing us about their results.) 

We can now obtain the Hamiltonian describing the or- 
bital interactions in the Ni layers of NaNiC>2 (see Fig. |l]a). 
In each layer Ni ions form a triangular lattice, the sites 
of which lie on intersections of the three sets of lines par- 
allel to the unit vectors e xy , e yz , and e zx , as shown in 
Fig. ||. It will be convenient to identify e Q( g with the three 
unit vectors in the isospace (see Eq.(||)) by n x = e yz , 
n y = e zx , and n z = e xy . The bonds between the nearest- 
neighbor sites j and j + e a p of the triangular lattice (the 
lattice constant is put to I) are diagonals of the Ni-0 
plaquettes lying in the a/3 plane (cf. Fig. |I]b). Hence, 
the Hamiltonian, describing orbital interactions on the 
triangular lattice has the form: 



Ht = JtY,Y, I P: 



a/3 



(7) 



are 



The terms linear in the operators Jj* (see Eq 
canceled in (0), since Y^ajf = ®- 

The Hamiltonian Eq.(ffl) is invariant under the global 




(a) 



(b) 



rotation of the isospin operators T a over the angle 



FIG. 3: The self-energy diagrams due to the qubic (a) and 
the quartic (b) interactions that result in a gap opening and 
a two-dimensional dispersion. 



around the y-axis combined with the rotation of the tri- 
angular lattice over the same angle. This transformation 
is equivalent to a cyclic permutation of the indices x, y, 
and z of the vectors and the isospin operators I a . 
Thus the Hamiltonian Eq.(|^) has the same symmetry as 
the so-called 'compass' model, describing exchange inter- 
actions between TM ions on a cubic lattice jj], [l3| (note, 
however, that in our case the three operators I x , I v , and 
I z are not independent). 

The orbital exchange on a triangular lattice turns out 
to be strongly frustrated and the mean-field ground state 
is not unique. The simplest states with the minimal 
energy Eq = — |j per site are ferro-orbital states, in 
which (Tj) = Tm on all lattice sites, where m is an 
arbitrary unit vector in the (T x , T z )-plane and T = ^. 
The energy of these states is independent of orientation 
of m, even though the Hamiltonian (|?]) is not invariant 
under arbitrary rotations in the (T x , T z )-plane (this con- 
tinuous ground-state degeneracy is similar to the one of 
the 'compass' model Q). Furthermore, there exist also 
disordered mean-field ground states, which can be ob- 
tained from the ferro-orbital states by inverting the sign 
of (T x ) on an arbitrarily selected set of lines parallel to 
e xy (see Fig. ^). Such states are ordered along the by- 
lines, but there are no long-range correlations between 
the x-projections of isospins in the transverse direction. 
By circular permutations of the x, y, and z indices one 
can obtain similar states, which are only ordered along 
the lines parallel to the e yz and e zx vectors. 

This large ground-state degeneracy is lifted by quan- 
tum orbital fluctuations (the so-called 'order-from- 
disorder' mechanism jl4], |ll|). First, we have checked 
numerically that disordered states (see Fig. ||) have a 
higher energy of the zero-point fluctuations than the cor- 
responding uniform states. Second, the zero-point energy 
of uniform states has 6 minima at the isospin orientations 
m = ±n x , ±n v , ±n z . In the original notation these are 
the ferro-orbital states with one of the orbitals d 3z 2_ r 2, 
d^ x 2_ r 2, d,3y2_ r 2, d x 2_ y 2, d y 2_ z 2, and d z 2_ x 2 occupied at 
each site. Furthermore, the anharmonicity effects |l6f| 
usually stabilize elongated octahedra, i.e. the orbitals 
d 3z 2_ r 2, d 3x 2_ r 2, or d 3y 2_ r 2. As was mentioned above, 
spins in layers order ferromagnetically. 

We would like to point out that the standard descrip- 
tion of orbital excitations with non-interacting bosons 
('isospin- wave approximation') is insufficient for the cal- 
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culation of quantum corrections to the ground-state en- 
ergy of the frustrated Hamiltonian Eq. ([?]). In this ap- 
proximation the orbital excitation spectrum for the six 
ground states is one-dimensional 



= 3V2TJ T 



sin 



(qm) 



(8) 



which is a direct consequence of the absence of long- 
range correlations between chains in disordered mean- 
field ground states. The gapless one-dimensional spec- 
trum leads to infrared-divergent fluctuations. To get rid 
of them, one has to take into account interactions be- 
tween the bosons, which can be consistently done in the 
large isospin limit, T 3> 1. The interactions suppress 
quantum fluctuations by opening a gap and inducing a 
dispersion in the direction transverse to m. This can be 
understood from the fact that the self-energy diagrams, 
shown in Fig. || (a) and (b), diverge unless a gap is intro- 
duced (in field theory a similar mechanism is known as 
a 'dynamical mass generation' [p"7| ). The physical origin 
of the gap is the breaking of the continuous ground-state 
degeneracy by quantum fluctuations |l|. In the large-T 
limit the quantum fluctuations are relatively small and 
the gap A cx JtVT is much smaller than the band width 
W ~ 3V2 J T T of the orbital excitations. The dispersion 
in the transverse direction is of the order of A |Q . 

The main conclusions of our theory of the exchange in 
frustrated systems (spins are coupled much weaker than 
orbitals, the ground state is ordered both ferro-orbitally 
and ferromagnetically) are in agreement with the or- 
bital and magnetic structure of the layered compound 
NaNi02, which undergoes two transitions: At T„ — 480K 
the oxygen octahedra become elongated, which corre- 
sponds to the ferro-orbital ordering of the d 3z 2_ r 2-type, 
and at the much lower temperature, T s — 20K, the Ni 
spins in the [111] layers order ferromagnetically ||. 

It is, therefore, very puzzling that the structurally 
identical material LiNi02 shows neither orbital nor spin 
ordering. In principle, we cannot exclude that the quan- 
tum orbital fluctuations, which for T = ~ are relatively 
large, destroy the long-range ferro-orbital ordering and 
stabilize an orbital liquid (such quantum melting of the 
'order-from-disorder' was discussed in the context of an- 
other frustrated system in Ref. fl9| ). Since for T = h 
there is no small parameter in the problem, the ques- 
tion whether the ground state of the orbital Hamiltonian 
Eq.(^) is ordered or disordered, can only be resolved by 
numerical calculations that are beyond the scope of this 
Letter. In any case, this possible explanation of the ab- 
sence of orbital ordering in LiNiC>2 is fundamentally dif- 
ferent from those based on a strong interplay between 
orbitals and spins {§ 0, Actually, due to the de- 
coupling of spins from orbitals in frustrated systems, an 
orbital liquid should still have a ferromagnetic order. 

We think that the puzzles surrounding LiNi02 indicate 
an importance of electron-lattice interactions and disor- 



der, not included in the present theory. As was argued in 
Refs. [||, |o|, the 'intrinsic' disorder caused by the pres- 
ence of magnetic Ni ions in the Li [111] planes, results in 
a strong interlayer coupling that frustrates the ferromag- 
netic spin ordering in Ni layers. In NaNiC>2 such disor- 
der should be weaker: as the size difference of Na and Ni 
ions is larger than that of Li and Ni ions, the alternating 
stacking of the Na and Ni [111] layers is more perfect than 
that of Li and Ni in LiNiC>2. The larger charge radius of 
the Na ion also implies stronger electron-lattice interac- 
tions in NaNiC>2. In general, such interactions suppress 
quantum orbital fluctuations and stabilize a ferro-orbital 
ordering [pll [22). This may explain why NaNiC>2 is or- 
bitally ordered, while LiNiC>2 is not. 

In conclusion, we developed a theory of the orbital and 
spin exchange in JT systems with doubly degenerate elec- 
tronic orbitals and 90° metal-oxygen-metal bonds. We 
showed that in these systems spins and orbitals are es- 
sentially decoupled: spins interact much weaker than or- 
bitals and the spin exchange is ferromagnetic for any 
orbital state. The orbital exchange between TM ions 
on a triangular lattice is strongly frustrated, resulting 
in infinite number of disordered classical ground states. 
An even stronger frustration can be shown to occur in 
spinels, in which TM ions form a pyrochlore lattice (to 
be published elsewhere). We showed, however, that small 
quantum orbital fluctuations remove the frustration and 
induce a ferro-orbital ordering. Our theory is an agree- 
ment with the orbital and magnetic structure observed 
in NaNiC>2. We also argued that large quantum fluctu- 
ations and 'intrinsic' disorder may, in principle, turn a 
frustrated system into an orbital liquid, which may ex- 
plain the absence of orbital ordering in LiNiC>2. 

We are grateful to A. Stepanov and to G. Chouteau 
and members of his group for extensive discussions of the 
experimental situation in LiNi02 and NaNi02, and to G. 
Khaliullin and L.-F Fcincr for the useful discussions. 



[1] 

[2] 
[3] 

[4] 

[5] 
[6] 

[7] 



[8] 
[9] 
[10] 

[11] 



K. I. Kugel and D. I. Khomskii, Sov. Phys. JETP 52, 
501 (1981); Sov. Phys. Uspekhi 25, 232 (1982). 
Y. Tokura and N. Nagaosa, Science, 5465, 462 (2000). 
S. Ishihara, M. Yamanaka, and N. Nagaosa, Phys. Rev. 
B 56, 686 (1997). 

L. F. Feiner, A. M. Oles, and J. Zaanen, Phys. Rev. Lett. 
78, 2799 (1997). 

B. Keimer et al. Phys. Rev. Lett. 85, 3946 (2000). 

G. Khaliullin and S. Maekawa, Phys. Rev. Lett. 85, 3950 

(2000). 

A. P. Ramirez, C. L. Broholm, R. J. Cava, and G. 
R. Kowach, Physica B 280, 290 (2000) and references 
therein. 

E. Chappel et al., Eur. Phys. J. B 17, 609 (2000). 
Y. Kitaoka et al., J. Phys. Soc. Jap. 67, 3703 (1998). 

F. Reynaud et al, Phys. Rev. Lett. 86, 3638 (2001). 

Y. Q. Li, M. Ma, D. N. Shi, and F. C. Zhang, Phys. Rev. 



■5 



Lett. 81 3527 (1998). 
[12] M. van der Bossche, P. Azaria, P. Lecheminant, and F. 

Mila, Eur. Phys. J. B 17, 367 (2000). 
[13] G. Khaliullin, Phys. Rev. B 64 212405 (2001). 
[14] J. Villain, R. Bidaux, J. P. Carton, and R. Conte, J. de 

Physique 41, 1263 (1980). 
[15] E. F. Shender, Sov. Phys. JETP 56, 178 (1982). 
[16] D. I. Khomskii and J. van der Brink, Phys. Rev. Lett. 

85, 3329 (2000). 
[17] S. Coleman and E. Weinberg, Phys. Rev. D 7, 1888 



(1973). 

[18] Details of the calculation of the dispersion of orbital ex- 
citations, which also requires a vertex renormalization, 
will be published elsewhere. 

[19] P. Lecheminant et al, Phys. Rev. B 56, 2521 (1997). 

[20] D. Mertz et al, Phys. Rev. B 61, 1240 (2000). 

[21] J. Kanamori, J. Appl. Phys., 31, 14S (1960). 

[22] R. Englman and B. Halperin, Phys. Rev. B 2, 75 (1970). 



